#gif number one and three are the same goal sequence

[gcsly]

[jcmarchi]

Differentially private median and more

New Post has been published on https://thedigitalinsider.com/differentially-private-median-and-more/

Differentially private median and more

Posted by Edith Cohen and Uri Stemmer, Research Scientists, Google Research

Differential privacy (DP) is a rigorous mathematical definition of privacy. DP algorithms are randomized to protect user data by ensuring that the probability of any particular output is nearly unchanged when a data point is added or removed. Therefore, the output of a DP algorithm does not disclose the presence of any one data point. There has been significant progress in both foundational research and adoption of differential privacy with contributions such as the Privacy Sandbox and Google Open Source Library.

ML and data analytics algorithms can often be described as performing multiple basic computation steps on the same dataset. When each such step is differentially private, so is the output, but with multiple steps the overall privacy guarantee deteriorates, a phenomenon known as the cost of composition. Composition theorems bound the increase in privacy loss with the number k of computations: In the general case, the privacy loss increases with the square root of k. This means that we need much stricter privacy guarantees for each step in order to meet our overall privacy guarantee goal. But in that case, we lose utility. One way to improve the privacy vs. utility trade-off is to identify when the use cases admit a tighter privacy analysis than what follows from composition theorems.

Good candidates for such improvement are when each step is applied to a disjoint part (slice) of the dataset. When the slices are selected in a data-independent way, each point affects only one of the k outputs and the privacy guarantees do not deteriorate with k. However, there are applications in which we need to select the slices adaptively (that is, in a way that depends on the output of prior steps). In these cases, a change of a single data point may cascade — changing multiple slices and thus increasing composition cost.

In “Õptimal Differentially Private Learning of Thresholds and Quasi-Concave Optimization”, presented at STOC 2023, we describe a new paradigm that allows for slices to be selected adaptively and yet avoids composition cost. We show that DP algorithms for multiple fundamental aggregation and learning tasks can be expressed in this Reorder-Slice-Compute (RSC) paradigm, gaining significant improvements in utility.

The Reorder-Slice-Compute (RSC) paradigm

An algorithm A falls in the RSC paradigm if it can be expressed in the following general form (see visualization below). The input is a sensitive set D of data points. The algorithm then performs a sequence of k steps as follows:

Select an ordering over data points, a slice size m, and a DP algorithm M. The selection may depend on the output of A in prior steps (and hence is adaptive).

Slice out the (approximately) top m data points according to the order from the dataset D, apply M to the slice, and output the result.

A visualization of three Reorder-Slice-Compute (RSC) steps.

If we analyze the overall privacy loss of an RSC algorithm using DP composition theorems, the privacy guarantee suffers from the expected composition cost, i.e., it deteriorates with the square root of the number of steps k. To eliminate this composition cost, we provide a novel analysis that removes the dependence on k altogether: the overall privacy guarantee is close to that of a single step! The idea behind our tighter analysis is a novel technique that limits the potential cascade of affected steps when a single data point is modified (details in the paper).

Tighter privacy analysis means better utility. The effectiveness of DP algorithms is often stated in terms of the smallest input size (number of data points) that suffices in order to release a correct result that meets the privacy requirements. We describe several problems with algorithms that can be expressed in the RSC paradigm and for which our tighter analysis improved utility.

Private interval point

We start with the following basic aggregation task. The input is a dataset D of n points from an ordered domain X (think of the domain as the natural numbers between 1 and |X|). The goal is to return a point y in X that is in the interval of D, that is between the minimum and the maximum points in D.

The solution to the interval point problem is trivial without the privacy requirement: simply return any point in the dataset D. But this solution is not privacy-preserving as it discloses the presence of a particular datapoint in the input. We can also see that if there is only one point in the dataset, a privacy-preserving solution is not possible, as it must return that point. We can therefore ask the following fundamental question: What is the smallest input size N for which we can solve the private interval point problem?

It is known that N must increase with the domain size |X| and that this dependence is at least the iterated log function log* |X| [1, 2]. On the other hand, the best prior DP algorithm required the input size to be at least (log* |X|)1.5. To close this gap, we designed an RSC algorithm that requires only an order of log* |X| points.

The iterated log function is extremely slow growing: It is the number of times we need to take a logarithm of a value before we reach a value that is equal to or smaller than 1. How did this function naturally come out in the analysis? Each step of the RSC algorithm remapped the domain to a logarithm of its prior size. Therefore there were log* |X| steps in total. The tighter RSC analysis eliminated a square root of the number of steps from the required input size.

Even though the interval point task seems very basic, it captures the essence of the difficulty of private solutions for common aggregation tasks. We next describe two of these tasks and express the required input size to these tasks in terms of N.

Private approximate median

One of these common aggregation tasks is approximate median: The input is a dataset D of n points from an ordered domain X. The goal is to return a point y that is between the ⅓ and ⅔ quantiles of D. That is, at least a third of the points in D are smaller or equal to y and at least a third of the points are larger or equal to y. Note that returning an exact median is not possible with differential privacy, since it discloses the presence of a datapoint. Hence we consider the relaxed requirement of an approximate median (shown below).

We can compute an approximate median by finding an interval point: We slice out the N smallest points and the N largest points and then compute an interval point of the remaining points. The latter must be an approximate median. This works when the dataset size is at least 3N.

An example of a data D over domain X, the set of interval points, and the set of approximate medians.

Private learning of axis-aligned rectangles

For the next task, the input is a set of n labeled data points, where each point x = (x1,….,xd) is a d-dimensional vector over a domain X. Displayed below, the goal is to learn values ai , bi for the axes i=1,…,d that define a d-dimensional rectangle, so that for each example x

If x is positively labeled (shown as red plus signs below) then it lies within the rectangle, that is, for all axes i, xi is in the interval [ai ,bi], and

If x is negatively labeled (shown as blue minus signs below) then it lies outside the rectangle, that is, for at least one axis i, xi is outside the interval [ai ,bi].

A set of 2-dimensional labeled points and a respective rectangle.

Any DP solution for this problem must be approximate in that the learned rectangle must be allowed to mislabel some data points, with some positively labeled points outside the rectangle or negatively labeled points inside it. This is because an exact solution could be very sensitive to the presence of a particular data point and would not be private. The goal is a DP solution that keeps this necessary number of mislabeled points small.

We first consider the one-dimensional case (d = 1). We are looking for an interval [a,b] that covers all positive points and none of the negative points. We show that we can do this with at most 2N mislabeled points. We focus on the positively labeled points. In the first RSC step we slice out the N smallest points and compute a private interval point as a. We then slice out the N largest points and compute a private interval point as b. The solution [a,b] correctly labels all negatively labeled points and mislabels at most 2N of the positively labeled points. Thus, at most ~2N points are mislabeled in total.

Illustration for d = 1, we slice out N left positive points and compute an interval point a, slice out N right positive points and compute an interval point b.

With d > 1, we iterate over the axes i = 1,….,d and apply the above for the ith coordinates of input points to obtain the values ai , bi . In each iteration, we perform two RSC steps and slice out 2N positively labeled points. In total, we slice out 2dN points and all remaining points were correctly labeled. That is, all negatively-labeled points are outside the final d-dimensional rectangle and all positively-labeled points, except perhaps ~2dN, lie inside the rectangle. Note that this algorithm uses the full flexibility of RSC in that the points are ordered differently by each axis. Since we perform d steps, the RSC analysis shaves off a factor of square root of d from the number of mislabeled points.

Training ML models with adaptive selection of training examples

The training efficiency or performance of ML models can sometimes be improved by selecting training examples in a way that depends on the current state of the model, e.g., self-paced curriculum learning or active learning.

The most common method for private training of ML models is DP-SGD, where noise is added to the gradient update from each minibatch of training examples. Privacy analysis with DP-SGD typically assumes that training examples are randomly partitioned into minibatches. But if we impose a data-dependent selection order on training examples, and further modify the selection criteria k times during training, then analysis through DP composition results in deterioration of the privacy guarantees of a magnitude equal to the square root of k.

Fortunately, example selection with DP-SGD can be naturally expressed in the RSC paradigm: each selection criteria reorders the training examples and each minibatch is a slice (for which we compute a noisy gradient). With RSC analysis, there is no privacy deterioration with k, which brings DP-SGD training with example selection into the practical domain.

Conclusion

The RSC paradigm was introduced in order to tackle an open problem that is primarily of theoretical significance, but turns out to be a versatile tool with the potential to enhance data efficiency in production environments.

Acknowledgments

The work described here was done jointly with Xin Lyu, Jelani Nelson, and Tamas Sarlos.

[ivystjamess]

𝐈 𝐃𝐎𝐍'𝐓 𝐏𝐋𝐀𝐘 𝐘𝐎𝐔𝐑 𝐑𝐔𝐋𝐄𝐒 𝐈 𝐌𝐀𝐊𝐄 𝐌𝐘 𝐎𝐖𝐍

WHO: @jazziejazxo​ and ivy st.james. ft mentions of: davis, julien, joey, kenna, noah, sammy, otto, lemon, ruby, eli, and winnie. WHEN: fri night. 2/5. WHERE: north hills mall. WHAT: bumping into each other at the mall, jaz and ivy have a heated exchange turned dream sequence of p!nk’s ‘cuz i can leading up to new direction’s regionals competition.

IVY LIKED TO THINK SHE HAD THREE HOMES. her actual one, whatever stage she be performing on (whether that be field or actual stage) and north hills mall. since entering her teenage years, she’d spent about as much time at the mall as she did doing her morning and nightly routines. so naturally, she knew the busy and the less busy times to go on her bi-weekly shopping sprees. thursday night was always pretty mild, and ivy liked it that way. she didn’t want to deal with a bunch of other people while shopping, who would? as she stepped onto the escalator to head down to the main floor, she caught the unmistakable gaze of jaz evans standing by the fountain and glaring daggers up at her. at this realization, ivy expedited her time on the escalator by walking down to give jaz a piece of her mind.

jaz’s intentions weren’t malicious at first. part of jaz’s goal for this semester was to refocus on the things that really matter. She was helping ND win regionals, being the best possible friend/sister/daughter, and checking drama at the door. after spending an entire weekend in bed and in constant worry that her voice wouldn’t recover from her random weekend sickness, jaz was more focused on these goals than ever. she was at the mall searching for the perfect gift to thank her mom for both taking care of her all weekend and helping her prep vocally for her big competition debut. but when she spotted ivy, all of that focus went out of the window. suddenly she was feeling just as insecure as she did when vocal adrenaline stormed the auditorium. her gaze didn’t waver even as ivy picked up speed. who did she think she was?

stepping up to the other girl, ivy shrugged her purse up on to her shoulder and flashed the fakest of smiles, “jaz so funny to see you here!” she started mild before concluding with a blunt, “i like couldn’t help but notice that look you were giving me.” a signature ivy st.james condescending pout graced her lips before she continued, “all shook up from the fundraiser? or was it our performance last week?” she asked with the tilt of her head. relations between ivy and jaz hadn’t always been so tense, in fact, they had a pretty healthy relationship prior to the blow up in the green room at sectionals. jaz always encouraged ivy to maybe be a little kinder, and ivy encouraged jaz to trust her talent a little more. that was all well and good until jaz had made it personal. taking ivy’s spot as captain? doing that duet with julien on the ski trip? ivy figured maybe jaz was a little ruthless with all this effort clearly being put into making her angry or jealous and arguing with her. with all this pent up frustration toward her former friend, ivy had no problem letting venom flow off her tongue unrestrained. “well, i have to get going! good luck with like, buying new directions’ cheap-o costumes or whatever you’re doing here. hopefully you find something that can hide the inevitably horrible choreography you’re going to be doing tomorrow. bye!” and with that, ivy turned her back, ready to walk away.

in a few blunt and nasty words, ivy had somehow confirmed all of the misconceptions jaz had made about this girl. somehow this new ivy was so bitter about not getting her way at sectionals that she literally transferred schools to create a “aha i told you so” moment. jaz was suddenly extremely angry at ivy, more than just disappointed. could she have been the reason davis just stopped talking to her out of the blue? jaz’s mouth moved faster than her brain could process. “honestly, whatever, ivy. just storm out or something…it’s the only thing you’re good at.” jaz spoke, venom dripping off of her words. she rolled her eyes and turned around, mumbling. “how were we even friends?”

wondering if maybe she had been a little too harsh, ivy froze in her place. obviously she didn’t want things to be this way. but ivy made her bed and had no problem laying in it. sure, there were surges where she missed the team, or the little moments like going to the pumpkin patch with jaz. it was the same as she got her surges of missing julien. but those days were over and gone. now? they were at war. 

both still heated from the exchange, their backs still turned, suddenly rock music began sounding throughout the mall as the pair aggressively whipped around to face each other. speaking over the music, jaz began ‘Rock and Roll, Rock! And I drink more than you! And party harder than you do! And my car's faster than yours too!’ making disgusted vocalizations at each other complete with lots of eye rolls and aggressive expressions, ivy pushed past jaz and strutted to the bathroom as she sang ‘P.I.N.K. P.I.M.P I'm back again I know y'all missed me.’ 

as the bathroom door swung open, ivy was suddenly in a locker room, donning work out gear. in the locker room, ivy continued to sing as she sat down on a bench and began taping up her hands, ‘Yeah I talk shit just deal with it.’ hands now taped, ivy continued to sing hostilely, slamming locker doors shut as she passed, the final one slamming shut as she belted, ‘You can try and try you can't be me!’

at the chorus, ivy and jaz both sounded as they popped up in different locations. ivy could be seen doing a variety of activities such as jumping rope, doing laps around an empty boxing gym characterized by black and blue ropes and banners, as well as sit ups all while davis lingered in the background either timing her, egging her on, or sitting on her feet as she did her sit ups. jaz, on the otherhand, remained in a gym decorated similarly to the other one, but with red and white adornments. in work out gear similar to ivy’s jaz also did a number of training activities; push ups, speed punching a punching bag, and running up and down the stands while julien stood by encouragingly as her coach. as this training went on they sang in perfect harmony, “but it's alright, I don't give a damn, I don't play your rules, I make my own, tonight I'll do what I want 'Cause I can”

when the chorus came to an end, the focus was pulled to jaz coming up from a push up and looking directly forward and singing, “I know I'm rare, you stop and stare, You think I care, I don't You talk real loud, But you ain't saying nothing cool” rising to her feet, she passed julien and gave him a high five as she moved to grab a water bottle and continued singing. glistening with sweat and aggression in her eyes, she arrived to the end of her verse and dramatically dumped the contents of her water bottle over her head as she belted out, ‘You can try and try you can't be me!’

moving into the second chorus, ivy and jaz were suddenly in a filled arena full of cheering fans. in ivy’s corner, eli, winnie, kenna, and sammy could be spotted sporting their TEAM IVY shirts in the front row, while in jaz’s corner noah, otto, lemon, and ruby could be spotted wearing their TEAM JAZ shirts as they cheered for the impending match. cutting between ivy and jaz as they entered with davis and julien from their tunnels, their voices again joined as they eyed each other from across the arena and sang their angst out. 

jaz hopped into the actual arena first where joey stood center in a referee’s shirt, but ivy was close behind. they were both followed by their respective coaches. as the bridge arrived, joey called the competitors to the middle of the ring. ivy and jaz shrugged off their blue and red robes and made their way to the center of the ring. while joey soundlessly laid out the rules of the boxing match, ivy and jaz’s eyes narrowed as they stared each other down. jaz began singing ‘yeah I'm super thick, people say I'm much too chick’ now verbally competing, ivy tauntingly cut her off, ‘come and kiss the ring, you just might learn a couple things.’ they were equally matched though as jaz now jumped in with ‘i'm tryin' to school ya dogs--’ but was abruptly stopped by joey blowing his whistle (which sounded a lot like ruff ruff ruff ruff ruff ruff ruff) and signaling their conduct was disorderly. paying no mind to the ref, ivy got right back in on the action with ‘i'm your worst nightmare.’ followed by a ‘bring it we can take it there’ from jaz, then concluded with ivy belting ‘what are you scared?’ over jaz singing the chorus.

with the climax of the song behind them, the last two sets of the chorus were filled with cut scenes of ivy and jaz leaning up against the ropes and vocalizing individually, circling each other in the rink ready to strike, and retreating to their corners to talk to their coaches. occasionally, a fan or two got rowdy enough in the crowd that it cut to their equally as energized antics, but when all was said and done, ivy and jaz stood in front of the north hills mall fountain wearing their normal clothes, backs facing each other, and absolutely silent other than the water bubbling beside them.

eventually both seem to come to the conclusion, it wasn’t worth engaging with one another. it would only be trouble, and a stinging reminder of friendship lost. it was easier to say nothing at all. so ivy stuck her chin up in the air, and jaz pushed some hair behind her ears, and they carried on their separate ways. 

THE END.

[4colorrebellion]

4cr Plays - Pixel Puzzle Makeout League (Switch)

Hi, so, yes, this is a little awkward. It has been awhile, hasn’t it, since my last post on this blog? I’m sure you have many questions. 

“How are you?” “Did you catch Covid-19?” “Where is my money?”

Or, more likely, “Are you really popping in after a couple of months of inactivity to discuss a hybrid of Picross and a dating sim?” 

The answer to that, dear readers, is yes. Fuck, yes. I am a fiend for Picross - logic puzzles where you fill in a pixel image based on numeric clues (also known by nonograms, pixel puzzles, and other names). I dove immediately from replaying the fantastic Mario’s Super Picross (available for Switch Online subscribers in the SNES app - it is a must-play!) into the subject of this post, Pixel Puzzle Makeout League. 

We are truly living in the renaissance period for Picross. In the old days, you had to wait years between releases of the Nintendo published series. Here and there, you might find an obscure Japanese import to help fill the gap. However, the advent of downloadable games, indie development, and mobile platforms have created more than enough takes on the core concept to give Picross fiends the constant drip feed needed to stave off the shakes (I feel seen). In addition to a bunch of variants of classic Picross, we have Picross-with-colors (Pic-a-Pix Deluxe, Piczle Colors), Picross-with-layers (Depixtion), graphic adventures (Piczle Cross Adventure), RPGs (PictoQuest), and murder mysteries (Murder by Numbers). Now, Pixel Puzzle Makeout League (PPML) introduces romance to the mix.

If you have never played one of these games before, it is worth taking a step back and explaining the core concept. You start with an empty grid. Each row and column of the grid has a series of numbers on it. Your goal is, simply, to fill the correct cells in the grid to create an image. You can think of it as a form of “paint-by-numbers”, where you need to identify the right spaces to mark to form a pixel-art painting.

The numbers on a row or a column tell you how many spots need to be filled in. If you see a “5″, you know that a five-block sequence will need to be filled, while blocks before and after those five will remain unfilled. To figure out which five need filled, you need to look at the corresponding columns (if you are working on a row) and see if you can match the conditions placed by the numbers on each of those. You will often see multiple numbers, like “2 3″, indicating that there are multiple sequences that will be filled in. In this case, the row would have a sequence of two filled-in blocks, then later, a second sequence of three filled-in blocks. At least one unfilled cell must exist between the two sequences. Again, you need to use the other clues to identify where the two sequences lie in the row.

That’s about all there is to it. Identify the spaces to fill, mark off those you know will not be filled, and try to complete the picture as fast as you can without making any mistakes. PPML offers both touch and button controls, but I’d really recommend just using the buttons. I feel the same about all of these Picross games - the buttons just give you more precision. 

As pictures get larger, keeping track of all of the numbers becomes a tougher task. Even as a novice, you can probably complete 5x5 or 10x10 puzzles without breaking a sweat. As the puzzles grow, you may find it a little harder to keep track of everything. Still, try not to panic. A great thing about these puzzles is that you should never have to make a blind guess. There is always enough information to unravel the next spot - it just may take some searching to identify it.

Pixel Puzzle Makeout League mostly sticks to the traditional Picross format - eschewing twists like colors or layers that other games have applied. This is fine. Even as someone who has solved thousands of these puzzles, there are so many different images you can form that it is not like the format itself grows stale. The actual puzzle quality in PPML is solid - nothing that will drive you insane in terms of difficulty, but not too easy either. There is a satisfying quantity and variety of puzzles, and there are a ton of bonus puzzles you can solve on the size to unlock themes - or when you want to take a break from reading. 

No, the core “twist” in PPML is the dating sim side of the game. Really, like Murder by Numbers, PPML is a visual novel. The game is broken into a series of scenes, where you read though dialogue between the characters and puzzles are used to accentuate the drama. The story revolves around a group of “superheroes” - four people who have been imbued with supernatural skill at solving different forms of puzzles by a mysterious power. 

Our protagonist, Pixel Girl, is the newest recruit to the Puzzle League, and is able to solve pixel puzzles with finesse and grace. Sudoku is a shy gamer who just wants to get back to her MMO group. Leader Chess is the tactical “genius” of the group, while Crossword uses his endless reservoir of knowledge to solve mysteries. Finally, we have Piecea. Piecea is... Well... Piecea is a sentient puzzle piece from another planet. He? She? They are, naturally, really good at puzzles. 

At the beginning of the game, Pixel Girl is kidnapped by a mysterious villain, Villain, who seems quite intent on getting her to join him in his sinister plot. You escape, but the next day, a massive “puzzle tower” emerges over the city. Together with your fellow Puzzle Leaguers, you need to put a stop to Villain before he can begin his reign of terror. 

As you can imagine, based on this description, PPML is not the most serious of games. The story and dialogue are actually really fun and well-written. The game itself is full of jokes about the different genres of puzzle involved, and the entire thing is just the right mix of humor, charm, and the surreal to be entertaining. I mean, it’s a game about a bunch of puzzle-based superheroes, and you can date a sentient puzzle piece. How is that not amazing?

The actual “romance” portion of the game comes into play at points where the game splits into different paths you can take, where you pick one of the characters to tag along with. During these alternate paths, you get to know the character you picked and grow closer to them. A nice feature of this is that your choice is not binding - as the plot always converges back to a central path, you can feel free to explore the potential relationships with each or all of the characters. You also get different puzzles along each path, giving more incentive to see every conversation. I’m a fan of Sudoku, but - come on - I also had to see what happened with the alien puzzle piece. How could you not? Really, all of the characters are charming enough that it is worth going down all of their paths. 

In truth, I’m not generally into dating sims, or even really the romance elements of most games. They tend to be some mixture of cheese, creepy, or just badly written. Here, the writing is cute and clever enough that I like it. There is just the right blend of romance and sarcasm to make it all work. 

Pixel Puzzle Makeout League is fantastic. It is a fun and clever take on the Picross formula, effectively using puzzles to liven up the visual novel. The writing is fantastic - a bit metatextual, a bit tongue-in-cheak, and a lot of genuine charm. All of it presented with nice art and enough puzzles to keep you busy for quite a while. If you are into the general Picross drama, then PPML is absolutely worth a look.

Just watch out for that alien puzzle piece. 

A copy of Pixel Puzzle Makeout League was provided for this review.

Official Website

Nintendo eShop